Semantic Instrumentation of Virtual Environments for Training

Interactive virtual environments are useful tools for hands-on learning or rehearsing of procedural tasks. However, task training applications typically provide a constrained course of action for the learner, forcing them down a single specific solution path. We discuss an approach in which the virtual environment is semantically instrumented in order to allow for the tracking of and reasoning about open-ended learner activity therein. Our approach leverages ontology-based knowledge representation which allows for a structured and meaningful description of virtual objects and of the learner actions that may be performed upon them. This is facilitated by the association of specific ontological classes with geometric components of the objects which populate a training exercise. These classes, together with their attributes, relationships and rules, characterize the environment and user actions in a readily understandable manner. As a result, a training system is able to observe the learner activity, render an assessment of that activity, and provide meaningful feedback to the learner. We also present an authoring tool which allows content developers to semantically annotate three-dimensional models for such an environment. ABOUT THE AUTHORS Christian Greuel is a Senior Research Artist in the Computer Science Laboratory at SRI International, with a focus on visual computing. His areas of applied research include design of tools, techniques, and processes for representation of and interaction with virtual environments; web-based training applications; geographic visualization; geometric modeling; multi-modal data representation; and immersive audio. He earned an MFA in Performing Arts Design and Technology from the California Institute of the Arts. Grit Denker is a Program Director in the Computer Science Laboratory at SRI International. Her expertise is in automated reasoning applications using semantic technologies, ontologies, and rules; cognitive and policy-based systems; specification and verification of security, network and privacy policies in distributed systems; and novel human-machine interface technology. She has a Ph.D. in Computer Science from the Technical University of Braunschweig, Germany. Karen Myers is a Principal Scientist in the Artificial Intelligence Center at SRI International, where she conducts basic and applied research in intelligent systems. Her specific areas of expertise include autonomy, intelligent assistants, mixed-initiative planning and control, and programming by demonstration. In addition to being widely published, she has overseen several successful transitions of her research into use both by the U.S. Government and within commercial settings. She has a Ph.D. in Computer Science from Stanford University.